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This study identified student and school-level factors associated with student 
achievement on the Ontario Secondary School Literacy Test (OSSLT), an examination 
that includes a student questionnaire examining home literacy practices. Linked 
student and school contextual data enabled the use of hierarchical linear modeling to 
complete the analyses and examine both student and school level effects. Fourteen 
student and three school level variables were found to be associated with students' 
reading achievement and twelve student and two school level variables were 
associated with writing. Significant variations between schools were also found for 
students with individualized education plans, English as a second language, or 
previous eligibility on the OSSLT. 
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Dans cette etude, les auteurs ont identifie les facteurs eleves et niveaux scolaires 
associes au rendement scolaire dans le Test de competences linguistiques de 
TOntario, un examen qui comprend un questionnaire pour les eleves au sujet des 
habitudes en matiere de litteratie a la maison. Des donnees reliees aux eleves et au 
contexte scolaire ont permis I'utilisation d'une modelisation lineaire hierarchique en 
vue de completer les analyses et d'etudier les effets quant aux eleves et aux niveaux 
scolaires. Les auteurs ont etabli que quatorze variables ayant trait aux eleves et trois 
ayant trait aux niveaux scolaires etaient associees au rendement des eleves en matiere 
de lecture et que douze variables ayant trait aux eleves et deux ayant trait aux 
niveaux scolaires etaient associees a Tecriture. Des ecarts importants entre les ecoles 
ont egalement ete notes chez les eleves ayant des programmes pedagogiques sur 
mesure, inscrits en anglais langue seconde ou deja admissibles au Test de 
competences linguistiques des ecoles secondaires de I'Ontario. 

Mots cles : modelisation lineaire hierarchique, test de competences linguistiques, 
rendement scolaire, facteurs pedagogiques 
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Policy frameworks that include large-scale assessments have been 
constructed to make judgments about what students know or can do, 
provide appropriate student guidance, and, increasingly, measure school 
performance (Delandshere, 2001; Ryan, 2002). In Canada, provincial 
governments use the results to report on such interests and to support 
interventions intended to improve them. The use of such assessments for 
the purposes of system accountability has been a more recent trend 
(Firestone, Mayrowetz, & Fairman, 1998; Ryan, 2002). One expression of 
the accountability framework is a response to the fear of declining 
standards combined with a concern for inequalities in students' 
opportunities to learn (Mazzeo, 2001). A second expression of 
accountability is associated with responsibility and reform (Ontario 
Royal Commission on Learning, 1994). Within this framework, 
educational accountability requires the availability of accessible and 
interpretable information about program quality and efforts to improve 
or maintain it. Such educational accountability frameworks are 
predicated on the belief that assuring quality of student performance is a 
key priority and that accountability can be used to support educational 
reforms and monitor and enhance student progress and school 
performance. 

Given the emerging accountability frameworks, and the call for data- 
based decision-making in education, there is a complementary desire to 
use large-scale assessment data to provide information to guide 
instructional and policy decisions leading to system improvement and 
growth. These data can be a rich source of information that, with the 
appropriate analyses and interpretation, lead to better understanding of 
and support for the performance and quality of schools and education in 
Canada. 

There is a definite need to better utilize these existing data sets. 
Fiowever, the relationships of schooling to student achievement are 
complex. Factors in addition to curriculum and instruction - for 
example, student characteristics on entry, teacher and school traits, home 
and community characteristics— have been shown to have significant 
relationships to student achievement, and these relationships vary across 
grade level, school, and subject area (Fitz-Gibbon, 1998; Ma, 2001; 
Mandeville & Anderson, 1987; Rumberger, 1995; Willms & Kerckhoff, 
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1995). Canada has a long history of collecting information on student 
achievement of learning outcomes, as well as characteristics of students, 
schools, and communities; however, the anonymous and/or incomplete 
nature of the data have resulted in restricted analyses. Consequently, the 
statistical analyses have often been limited to examining correlations 
between educational indicators, (e.g., expenditures, school type) and test 
scores, with inconsistent results. Some researchers (e.g., Greenwald, 
Hedges & Laine, 1996a, 1996b) claim clear associations exist among 
various educational indicators whereas others (e.g., Hanushek, 1996) fail 
to find such associations. 

Contextual student variables, also associated with higher 
achievement, are important to consider when examining student 
achievement in the context of large-scale assessments (Ma & Klinger, 
2000; Taylor & Tubianasa, 2001; Willms, 1992). These include, but are not 
limited to, foreign language status, learning disabilities, family status, or 
special needs (e.g., Cummins, 1982; Hamp-Lyons, 1996; Ma & Klinger, 
2000; Rogers, Ma, Klinger, Dawber, Hellsten, Nowicki, & Tomkowicz, 
2000; Watt & Roessingh, 2001; Willms, 1992). Similarly, researchers have 
consistently identified indicators of wealth and education, at both the 
individual and school level, as important contextual variables affecting 
individual achievement (e.g., Caldas & Bankston, 1997; Raudenbush & 
Willms, 1995; Sirin, 2005; Willms, 1992). To increase educators' 
understanding of student and school performance and to support efforts 
to increase such performance, it is important to include and better 
understand the associations of such factors with measures of student 
achievement. 

Not surprisingly, researchers have found achievement related 
variables to be associated with subsequent achievement. Previous 
achievement, an involvement in activities related to the measure of 
interest, and access to relevant materials are all associated with higher 
achievement (e.g., Rogers, et ak, 2000, Willms, 1992). For literacy, the 
quantity and types of reading and writing activities have been linked to 
higher achievement although the majority of the researchers have 
examined early literacy development (e.g., Leseman & de Jong, 1998; 
Payne, Whitehurst, & Angell, 1994; Saracho, 1997; Scarborough, Dobrich, 
& Hager, 1991). Relevant computer use has also been linked to higher 
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achievement; however, these associations are much less definitive (e.g., 
Attewell & Battle, 1999; Miller & Mclnerney, 1994; Ravitz, Mergendoller, 
& Rush, 2002). These associations have been found af bofh fhe individual 
sfudenf level and fhe school level. Hence sfudenfs who attend schools in 
which more sfudenfs engage in supporfing acfivifies will also be more 
likely fo have higher achievemenf (e.g., Ma & Klinger, 2000; Willms, 
1992). Such research indicafes fhaf achievemenf is relafed nof only fo fhe 
efforfs and acfivifies of individual sfudenfs buf also fhe efforfs and 
acfivifies of schools and fheir sfaffs. 

However, if is nof clear where research efforfs should be direcfed. 
Previously used analyfic procedures have been unable fo address such 
issues. For example, evidence suggesfs fhaf fhe level af which fhe sysfem 
fraifs are aggregafed influences fhe predicfabilify of resulfs. Yair (1997) 
poinfed ouf fhaf wifhin-school variafion in sfudenf achievemenf is 
generally greafer fhan befween-school variafion. The befween-school 
variafion is furfher complicafed by fhe nesfed sfrucfure of educafion 
because sfudenfs are nesfed wifhin classrooms (and feachers) fhaf, in 
fum, are nesfed wifhin schools. Tradifional sfafisfical procedures can be 
negafively affecfed by such nesfed dafa because fhe classroom and 
school effecfs differenfially impacf sfudenf performance. These nesfed 
dafa lend fhemselves well fo mulfi-level or hierarchical linear modeling 
(HLM). HLM facilifafes fhe examinafion of relafions occurring af each 
level, across levels (specify how variables af one level influence relafions 
occurring af anofher), and assesses fhe amounf of variafion af each level 
(Raudenbush & Bryk, 2002; Snijders & Bosker, 1999). Mulfi-level analyses 
offer a useful alfemafive for nesfed dafa because fhey beffer accounf for 
fhe non-independence of observafions wifhin groups and are more 
aligned wifh fhe fradifion of model-based inferences (Raudenbush & 
Bryk, 2002; Snijders & Bosker, 1999). 

The analyses of dafa arising from large-scale assessmenf programs 
have been limifed eifher by fhe nafure of fhe examinafion, fhe dafa- 
collecfion design, or fhe lack of confexfual sfudenf dafa. Many large-scale 
assessmenf programs used in previous research combine a low-sfakes 
assessmenf program (e.g., PISA, TIMMS, SAIP) wifh sfudenf and, af 
fimes, feacher and/or school surveys. Because of fhe low-sfakes of fhe 
fesf, if can be argued fhaf sfudenfs are less mofivafed to produce their 
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best work, teachers are not motivated to encourage maximum student 
performance, and not all survey items are answered, leading to problems 
with data quality or missing data. Research using data from fhese 
assessmenf programs is furfher hampered by dafa collecfion mefhods. 
Alfhough fhese assessmenf programs use a large randomly selecfed 
sample of sfudenfs, fhe number of sfudenfs chosen from a particular 
school is generally quite small. Hence it is difficult to identify specific 
school effecfs associafed wifh achievemenf. In confrasf, several 
educafional jurisdicfions use high-sfakes assessmenf programs. For 
example, in Canada, Brifish Columbia, Alberfa, and Quebec use high- 
school provincial examinafion programs fo help defermine high-school 
grades and graduafion. The majorify of fhe eligible sfudenfs wifhin a 
school complefe fhese assessmenfs. However, fhese assessmenf 
programs have nof included sfudenf, feacher, or principal surveys. 

PURPOSE 

In confrasf fo ofher large-scale fesfing programs, fhe Onfario Secondary 
School Liferacy Tesf (OSSET) program includes a high-sfakes fesf along 
wifh confexfual and achievemenf relafed sfudenf dafa obfained from fhe 
school fhrough fhe sfudenf informafion form and a brief sfudenf survey. 
As well, pofenfially relevanf school informafion can be obfained fhrough 
amalgamafion of fhe sfudenf level dafa and from ofher sources, for 
example, fhe Educafional Qualify Indicafor Framework (EQI). The 
purpose of fhe currenf sfudy was fo idenfify whaf sfudenf and available 
school-level factors were associated with student achievement on the 
OSSET, a high-stakes examination of sfudenf liferacy. Given fhe nesfed 
sfrucfure of fhe dafa (sfudenfs nesfed wifhin schools), HEM was used fo 
avoid fhe problems associafed wifh simple correlafional analyses and 
mulfiple regression fechniques. 

The Ontario Secondary School Literacy Test 

The OSSLTi is a graduafion requiremenf for high-school sfudenfs in 
Onfario. Implemenfed in 2002, fhe OSSET is fhe firsf high-sfakes 
examinafion in Ontario schools that includes relevant student contextual 
information and a student survey about student literacy activities. The 
purpose of fhe OSSET is fo ensure fhaf sfudenfs have acquired fhe 
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essential reading and writing skills for understanding reading selections 
and communicating in a variety of wrifing forms faughf up fo fhe end of 
grade 9 and fhaf apply fo all subjecf areas in fhe Onfario curriculum 
(Educafion Qualify and Accounfabilify Office [EQAO], 2005a, 2005c). As 
sfafed by EQAO (2005b), fhe OSSET defermines "whefher individual 
sfudenfs have mef fhe liferacy graduafion requiremenf" (p. 3). In 
fulfilling ifs accounfabilify mandafe, a sfudenf's success on fhe OSSET 
gives "fhe public confidence fhaf fhe sfudenf has acquired fhe basic 
reading and wrifing skills expecfed by fhe end of grade 9" (EQAO 2005c, 
p. iv). As parf of ifs mandafe, EQAO also promofes fhe use of fhe OSSET 
results to support classroom instruction and raise student achievement. 

All students in public and private Ontario secondary schools who 
are working toward an Ontario Secondary School Diploma are required 
to successfully complefe the OSSET or the corresponding Ontario 
Secondary School Literacy Course (OSSLC) prior to graduation (Ontario 
Ministry of Educafion, 2003). Sfudenfs regisfered in fheir second year of 
high school (fypically grade 10) are eligible fo wrife fhe tesf, alfhough 
fhey may be deferred from wrifing fhe assessmenf unfil a subsequenf 
adminisfrafion. The OSSET can be wriffen in eifher English or Erench 
depending on fhe school board in which fhe sfudenf is enrolled. The tesf 
design from its inception in 2002 until the 2004 administration consisted 
of fwo 2.5-hour componenfs^ - reading and wrifing in a prinf confext. 

The following informafion, associafed wifh fhe 2003 Ocfober 
adminisfrafion, was used in fhe currenf sfudy.^ In fhe reading 
componenf, sfudents complefed a fofal of 12 reading selecfions which 
were classified as informafion (e.g., wriffen explanafion, opinion; worfh 
50%), graphic (e.g. graph, schedule, insfrucfions; worfh 25%), and 
narrative (e.g., story, dialogue; worfh 25%). Three differenf quesfion 
formafs were used fo measure sfudenfs' comprehension and 
undersfanding of fhese selecfions: multiple-choice (MC) (40 quesfions; 
worfh 40%), quesfions requiring a shorf construcfed response (CR) (35 
quesfions; worfh 35%), and quesfions requiring a consfrucfed response 
wifh an explanafion (CRE) (25 quesfions; worfh 25%). In fhe wrifing 
componenf, sfudents completed four fasks: a summary, a series of 
paragraphs expressing an opinion, a news report, and an information 
paragraph. Eor each task, students were instructed to organize their 
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ideas, write complete sentences, and use standard spelling, grammar, 
and punctuation. The test provided students with the purpose and 
audience for each task and guidelines for fhe lengfh and sfrucfure of 
fheir wrifing samples. 

The completed tests were sent to EQAO and centrally marked by 
trained teachers and markers. The MC and CR items on the reading 
component were scored on a 2-point (0, 2) scale and the CRE items were 
scored using item specific scoring rubrics using a 3-poinf scale (2 poinfs 
for correcf, 1 poinf for parfly correcf, or 0 for incorrecf). An equafed 
scaled score of 125/200 represenfed fhe minimum passing score for 
reading. Scores befween 115 and 124 were rescored. Each wrifing fask 
was scored using a 4-poinf (10, 20, 35 and 45) scoring scale having 
specific performance descriptors. A score of 0 was given for responses 
classified as "blank/illegible and irrelevanf confenf/off-task." Sfudents 
who obfained af leasf 100/180 received a "pass" on fhe wrifing 
componenf. A separate buf single rater scored a sfudenf's four wrifing 
fasks. However, sfudenfs who failed fhe wrifing tesf buf were wifhin 10 
points (e.g., between 90 and 100) of fhe passing score were rescored by a 
second sef of rafers. For fhese sfudenfs, fhe highesf sef of scores was used 
for fhe wrifing score. 

Sfudenfs who wrote fhe OSSET were informed fhaf fhey had eifher 
successfully or unsuccessfully complefed fhe fesf. Unsuccessful sfudenfs 
also received an Individual Sfudenf Reporf (ISR) fhaf provided defails 
abouf fheir performance wifh fhe various fypes of reading maferials, 
specific reading skills, and fhe four wrifing fasks. 

Af fhe same fime sfudents completed the OSSET, they also 
completed a student survey that asked for information regarding home 
liferacy practices. The survey was designed fo be complefed wifh little 
exfra efforf or fime, using largely dichofomous ifems. 

METHOD 

Sample 

The sample for fhe currenf study consisted of sfudenfs who complefed 
fhe English version of fhe OSSET in Ocfober 2003. Separate analyses 
were complefed for fhe reading and wrifing componenfs. Data were 
obtained for an inifial sample of 218,242 sfudenfs. Of fhese, 192,457 
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students completed the English reading and writing components and 
180,670 students completed the accompanying survey. List-wise deletion 
of missing data reduced the student sample to 177,638 students with 
complete student data. The sample consisted of sfudenfs who were 
wrifing fhe fesf for fhe firsf fime as well as fhose who eifher previously 
failed fo successfully complefe or were deferred from fhe assessmenf. 
Schools wifh fewer fhan 10 sfudenfs or fhose schools fhaf did nof have 
relevanf EQI dafa were removed from fhe sample resulfing in a final 
sample of 160,491 sfudenfs nesfed wifhin 611 public and cafholic schools 
(EQI dafa was nof available for privafe schools). 

Data 

The dafa for fhe analyses came from fwo sources. Sfudenf level 
achievemenf scores, demographic informafion, and liferacy acfivify dafa 
came from fhe OSSET and accompanying sfudenf survey. Through 
identification codes, if was possible fo link individual sfudenf OSSET 
scores and survey responses. Sfudenfs were also linked fo fheir 
individual schools fhrough school identification codes. These codes 
defermined average school values for fhe sfudenf survey ifems. The EQI 
dafabase was used fo obfain an esfimafe of fhe average income of 
families in each school. This informafion was based on fhe 2001 census 
dafa using fhe posfal codes of families in fhe school rafher fhan in fhe 
neighborhood around fhe school. Elence if likely provided a more 
accurafe measure of fhe average socio-economic sfafus of fhe sfudenfs 
wifhin fhe schools. The number of firsf-fime eligible sfudenfs wrifing fhe 
assessmenfs was used as an esfimafe of school size. The school board 
names were used fo classify schools as eifher public or cafholic schools. 
Classification of urban or rural location was nof available in fhe dafasefs 
and was nof included in fhe analysis. Affempfs were made fo include an 
education variable using fhe EQI dafabase; however, fhe correlafions 
befween fhe income and fhe education variable (0.93) indicafed fhaf a 
single indicafor would be appropriafe. Given fhaf fhe income variable 
was mosf associafed wifh fhe school population, fhaf variable was used. 
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DATA ANALYSIS AND RESULTS 
Factor Analysis 

Student level. To reduce the number of variables included in the analyses, 
exploratory factor analyses were conducted to identify sets of variables 
that could be used to form composite factors. Principal component 
analysis with varimax rotation of the components with eigen-values 
greater than or equal to one was used in the analysis. Using only those 
items with single loadings of at least 0.30, two factors were created. The 
scores for these factors were calculated using the sum of the items within 
each factor (Morris, 1979). The first factor, entitled creative literacy 
activities, included four questions from the student survey that asked 
about literacy related activities that the students did outside of school: 
read letters (31% of students responding yes), read poetry (41% 
responding yes), write letters (36% responding yes), and write songs 
(32% responding yes). The second factor, home materials, included the 
four questions that asked students to identify which of the following 
reading/literacy materials they had at home: books (89%), magazines 
(85%), dictionaries (86%), and newspapers (85%). The reliabilities 
(Cronbach's a) of these two scales were 0.56 and 0.68, respectively. These 
factors were used rather than the individual items in the HLM analyses. 
Because of the lack of clear factor loadings, the other literacy related 
variables were maintained as discrete predictors in the analyses. 

School level. Factor analysis was also used to determine if any of the 
school average student survey responses could be reduced into 
interpretable school level factors. Based on this analysis, three factors 
were created. The first, school average home materials, consisted of four 
variables: percentage of students reporting having books, magazines, 
dictionaries, and newspapers at home. The internal consistency 
(Cronbach's a) of this factor was 0.86. The average creative literacy 
activities factor also contained four variables: percentage of students 
who read letters, read poetry, write letters, and write songs outside of 
school. This factor had an internal consistency of 0.83. The two average 
internet use variables (read on the internet and reading email) formed a 
doublet factor, average internet use. This factor had a Cronbach's alpha 
of 0.91. 
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Descriptive Results 

Table 1 provides the descriptive statistics for the data used in the 
analyses. For aggregated student variables, the school and student level 
variables had the same observed means. Consequently, only the 
standard deviation for schools is reporfed in fhe final column of Table 1. 
In fhe case of dichofomous variables, fhe sfandard deviafions are nof 
provided for the student level variables. Unless otherwise noted, the 
student status variables were dichotomously coded (0, 1), with "1" 
signifying designafion of each variable. The sfafus variables were official 
provincial designafions for specific sfudenfs as idenfified by fhe school. 
In Onfario, English as Second Language and English Liferacy 
Developmenf (ESL/ELD) learners are combined as a subgroup of 
recenfly immigrafed sfudenfs. ESL sfudenfs can read and wrife in a firsf 
language (ofher fhan English) and mosfly have had confinued schooling 
before arriving in Canada. ELD sfudenfs are fhose who may nof read 
and/or wrife in fheir firsf language (LI) and may have missed years of 
schooling. They could come from counfries where Sfandard English is 
fhe official dialed buf where ofher dialecfs of English are in common use 
(Onfario Minisfry of Education and Training, 1999). 

Sfudenfs wifh an individual education plan (lEP) due fo special 
needs or required learning supporf or wifh an idenfificafion placemenf 
and review committee (IPRC) designafion ofher fhan gifted were 
classified under fhe cafegory of lEP for fhe analyses. These sfudenfs 
included fhose wifh learning disabilifies, physical disabilities, or 
behavioural problems. Gifted sfudenfs were idenfified via fhe IPRC and 
given an "IPRC-giffed" designafion. Finally, sfudenfs who had 
previously been deferred from writing fhe OSSET, were absenf, or 
unsuccessful on a previous affempf were considered "previously 
eligible." Two ofher variables fhaf described sfudenf sfafus relafed fo 
home language were included in fhe sfudenf survey quesfiormaire. In 
fhe firsf quesfion, students were asked to identify if fhe firsf language 
fhey learned af home was English or nof. These sfudenfs were 
considered fo be fhose for whom English was a foreign language (EFL). 
The second quesfion asked sfudenfs fo describe fhe language fhey spoke 
af home, only or mosfly English, anofher language as often as English, or 




Contextual and School Factors 


781 


Table 1: Descriptive Statistics for Study Variables and Factors 


Students 
(n = 160,491) 

Schools 
(n = 611) 


Reading score 

143.33 (31.75) 



Writing score 

116.02 (41.19) 



Student Status (% of students) 




Gender (female = 1) 

48 


(9) 

ESL status 

3 


(6) 

lEP status 

15 


(15) 

IPRC-gifted 

1 


(2) 

Previously eligible 

26 


(15) 

English as a foreign language (EEL) 

22 


(20) 

Foreign home language 

23 


(20) 

Home Literacy Survey Items (% of students) 




Read comics 

32 


(6) 

Read non fiction 

24 


(5) 

Read on the internet 

77 


(8) 

Read novels 

52 


(9) 

Read newspapers 

46 


(8) 

Read manuals 

21 


(4) 

Read magazines 

73 


(6) 

Read religious materials 

14 


(6) 

Writing email 

88 


(7) 

Writing notes 

34 


(5) 

Writing stories 

17 


(6) 

Writing work 

36 


(6) 

Scaled Survey Items and Factors 




Reading hours per week 

2.15 (1.04) 


(0.18) 

Writing hours per week 

1.99(1.01) 


(0.12) 

Home computer use for school work. 

3.74 (1.08) 


(0.35) 

Home literacy materials 

3.39 (1.01) 


(0.23) 

Creative literacy activities 

1.42 (1.34) 


(0.20) 

Variables and Factors reported at the school level only 




Number of Students writing the 


263 (121) 
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OSSLT 
School Type 

Students' average Internet use 
Mean Family Income 

Notes: The values in parentheses are standard deviations. 

The % of students and average school % of students are equal. 

only or mostly another language. Because of the small percentage of 
sfudenfs choosing opfions 2 or 3, 14.4 per cenf and 6.9 per cenf 
respecfively, fhis "Foreign home language" variable was fransformed 
info a dichotomous variable (0, 1) in which a score of 1 indicafed fhaf fhe 
language commonly spoken af home was nof English. 

The sfudenf home liferacy dichotomous survey variables (0, 1) were 
coded such fhaf a "no" response was given a score of zero and a "yes" 
response was given a score of one. Lasfly, fwo of fhe fhree scaled survey 
fimes - reading hours per week and wrifing hours per week - were 
measured on a 4-poinf scale from 1 fo 4, while fhe fhird - home 
compufer use for school work - was measured on a 5-poinf scale (1 fo 5). 
The lasf fwo sfudenf variables were fhe fwo facfors idenfified fhrough 
factor analysis. 

Comparison of fhe samples used in fhe presenf sfudy fo fhe overall 
sfudenf sample indicafed fhaf fhe reduced, analyzed sfudenf sample was 
very similar fo fhe acfual full sample. The reading and wrifing scores 
were slighfly higher in fhe analyzed sample (less fhan 2 score poinfs) and 
fhe percenfages of ESL and lEP sfudenfs were slighfly lower. All four of 
fhese variables also had slighfly smaller sfandard deviafions in fhe final 
sample. 

The four remaining variables reporfed af fhe boffom of Table 1 were 
collecfed solely af fhe school level. The number of sfudenfs wifhin a 
school who were eligible for fhe firsf fime and complefed fhe OSSLT 
during fhe 2003 adminisfrafion of fhe fesf is lisfed firsf. The second 
variable, "school type" was used to distinguish between public and 
catholic schools, both of which are fully funded in Onfario. A fofal of 24 
per cenf of fhe schools included in fhe analyses were considered fo be 
cafholic schools. 


24 

1.62 ( 0 . 13 ) 
59341 ( 11776 ) 
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HLM Analysis 

Before the analyses of the set of student and school level variables, a 
"null" model was examined containing no explanatory variables. This 
model indicates the amount of variability at each level of analysis. With 
respect to reading, 86.5 per cent of the variability in reading scores was 
found to exist between students and just over 13.5 per cent of the 
variability in reading scores occurred between schools. The between 
student variability was even higher for writing scores (90.3%), with the 
variability between schools being 9.7 per cent. Because class effects were 
not measured, it was not possible to separate student and class effects. 

HLM models were then examined to determine which constellation 
of student and school level variables best predicted reading and writing 
achievement on the OSSLT. All the Level 1 variables identified in Table 1 
were included in the HLM analyses. Student level variables were also 
examined to determine if the effects were constant across schools (fixed 
slope effects) or different across schools (identified by HLM as random 
[differential] slope effects). Fixed slope effects indicated a relationship 
between a student level explanatory variable and achievement but the 
relationship was constant within and across schools. Student level 
explanatory variables were not fixed in the analyses if the between 
school variances (tau) because the variables were considered potentially 
meaningful based on the criterion that the variance was significant (p < 
0.01) and greater than 0.05. Such variables do not have consistent 
associations with the dependent variable between schools. Differential 
(random) school effects can be important from a research and policy 
perspective because they demonstrate that an association may be 
susceptible to modification among schools. For student level variables 
having differential school effects, school level variables may help explain 
why a "gap" is present in some schools but not in others. Although 
generally providing a superior explanatory model, the disadvantage of 
the inclusion of such differential effects is the difficulty in determining 
the amount of variance accounted for in the final model. However, in the 
presence of differential effects, Snijders and Bosker (1999) note that an 
estimate of the accounted variance can be found by running the final 
model as a fixed model. All non-dichotomous student scores were 
transformed into z-scores for the purposes of HLM analyses. 
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Consequently, because all the scores were on a 0/1 metric for the 
dichotomous variables and a z-score for non-dichotomous variables, fhe 
coefficienfs indicafed fhe relafive change in fhe dependenf variable fhaf 
was associafed wifh a unif change in fhe explanafory (independenf) 
variable. 

School level variables were examined affer including fhe sef of 
significanf studenf level variables. School level variables indicafed fhe 
impacf of fhe average school effecf on sfudenfs' dependenf variable 
scores wifhin fhaf school affer accounfing for relevanf sfudenf factors. In 
fhe case of sfudenf level variables having differenfial (random) slopes, 
significanf school level variables were fhose fhaf accounfed for a porfion 
of the identified variance in fhe relationship befween a sfudenf-level 
explanafory variable and fhe dependenf variable across schools. Because 
school effecfs provide a measure of fhe overall impacf across all sfudenfs 
in a school, fhey are generally smaller fhan sfudenf-level effecfs. Wifh fhe 
exception of reading novels, fhe individual dichotomous survey items 
aggregafed to the school level were not included in the HLM analyses. 
This decision was made because of fhe low variability of fhese variables 
af fhe school level and fheir lack of impacf af Level 1 in fhe presence of 
ofher sfudenf level explanafory variables. All school level variables were 
fransformed info z-scores. Hence fhe coefficienfs indicated the relative 
change in the average achievement score for a school (infercepf) for 
every sfandard deviafion change in fhe explanafory variable. 

The resulfs of fhe HLM analyses are provided in Table 2 for fhe 
sfudenf-level variables and Table 3 for fhe school-level variables. Only 
fhose variables fhaf were significanfly associafed wifh achievemenf and 
had a coefficienf of af leasf 0.05 in absolufe value were included in fhe 
final model. This would eliminate any variables fhaf may have a slighf 
associafion with student achievement but whose effect would be too 
small to have a meaningful impacf on sfudenf achievemenf. The resulfs 
for reading are presented in fhe second column of each fable and fhe 
resulfs for wrifing are presented in fhe fhird column. The coefficienfs 
reported for fhe predicfive variables included in fhe model are adjusfed 
for fhe presence of fhe ofher predicfive variables included in fhe model. 
For example, fhe coefficienf for ESL sfafus for reading achievemenf, 
-0.43, has been adjusfed for fhe ofher predictor variables included in the 
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model. This value suggests that after controlling for fhe remaining 
variables, ESL sfudenfs on average had a reading score approximafely 
0.43 of a sfandard deviafion below fhe mean reading score for fhe full 
sample of sfudenfs. 


Table 2: Coefficients for the Student Level Variables 


Student Variables 

Reading 

Writing 

Intercept 

0.02 (<0.01) 

0.08 (<0.01) 

Student Status 



Gender 

0.05 (<0.01) 

0.09 (<0.01) 

ESL status 

-0.43 (0.02) 

-0.07 (0.03) 

lEP status 

-0.40 (0.01) 

-0.28 (0.01) 

IPRC-gifted 

0.52 (0.01) 

0.37 (0.02) 

Previously Eligible 

-0.51 (0.01) 

-0.97 (0.02) 

EEL 

-0.05 (<0.01) 


Foreign home 
language 

-0.12 (<0.01) 

-0.07 (<0.01) 

Home Literacy 



Read on the internet 

0.16 (<0.01) 

0.10 (<0.01) 

Read novels 

0.18 (<0.01) 

0.13 (<0.01) 

Scaled Survey Items and Factors 



Reading hours/week 

0.13 (<0.01) 

0.09 (<0.01) 

Home computer use 
for school work. 

0.05 (<0.01) 


Home materials 

0.12 (<0.01) 

0.07 (<0.01) 

Creative literacy 

activities 

-0.08 (<0.01) 

-0.06 (<0.01) 


Note: The numbers in parentheses represent the standard error of estimate. 


Reading results. As shown in Table 2, fhe seven sfudenf sfafus 
variables, fwo of fhe 12 home liferacy survey ifems, and four of fhe five 
scaled survey ifems/facfors were included in fhe model for reading af fhe 
sfudenf level. Wifh one excepfion, fhe associafions befween fhese 
variables and reading achievemenf were in fhe expecfed direction. Four 
of fhe seven variables included in fhe sef of sfudenf sfafus variables had 
fhe largesf relafive associafions wifh reading scores. The relafively large 
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negative associations of ESL status, lEP status, and previously eligible 
status with reading achievement indicated that these students' reading 
scores were, respectively, on average 0.43, 0.40, and 0.51 of a sfandard 
deviafion lower fhan fhe average reading score. In confrasf, IPRC-giffed 
sfudenfs had reading scores fhaf were approximafely a half of a sfandard 
deviafion (0.52) higher fhan average reading score. The associafions of 
fhe remaining fwo sfudenf sfafus variables were small (-0.05 and -0.12). 
Two dichofomously scored home liferacy variables, reading on fhe 
infemef and reading novels, had relafively small buf posifive 
associafions wifh reading achievemenf, 0.16 and 0.18 respecfively. The 
associafions of fhe four significanf scaled variables were relafively small. 
Of fhese variables, fhe number of reading hours per week and home 
liferacy maferials had fhe highesf posifive associafions, 0.12 and 0.13, 
respecfively. Inferesfingly, fhe associafion befween engagemenf in 
creafive liferacy acfivifies was negafively associafed (-0.08) wifh reading 
fesf performance. 

The school level resulfs for fhe reading model are presenfed in fhe 
second column of Table 3. Panel A confains fhe school-level relafionships 
wifh fhe school mean (infercepf). Panel B confains fhe resulfs for fhe 
fhree sfudenf-sfafus variables - ESL (tau = 0.11), lEP (tau = 0.06), and 
previously eligible (tau = 0.09) - fhaf had significanf befween school 
variance, suggesfing fhaf fhe relafive differences in reading scores 
obfained by fhese designafed sfudents as compared fo fhe ofher sfudenfs 
in fhe school were nof consisfenf across schools. Thaf is, fhe gap befween 
fhe average performance of ESL and non-ESL sfudenfs was larger in 
some schools fhan in ofhers. Similarly, fhese gaps varied across schools 
for lEP and previously eligible sfudenfs. Elence relafive differences in 
reading performance could be affribufed, af leasf in parf, to the schools 
that these students attended. 

Three Level 2 variables were found fo have significanf associafions 
wifh reading achievemenf in fhe final model (Panel A, Table 3). The 
coefficienfs were all weak, ranging from 0.05 (average reading hours per 
week) fo 0.12 (average home liferacy maferials). As fhe average use of 
home liferacy maferials, hours spenf reading, and use of fhe infemef 
increased, school reading performance increased. 
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Table 3: Coefficients for the School Effects 


Variable 

Reading 

Writing 

Panel A: School Mean 

Average reading hrs/wk 
Average home computer 
use for school work 

Average home materials 
Average Internet use 

0.05 (<0.01) 

0.12 (0.01) 

0.06 (<0.01) 

0.10 (0.01) 

0.09 (<0.01) 

Panel B: Differential Slopes 

ESL slope 

School lEP 

Number of students 
Previously eligible 

0.11 (0.04) 

-0.07 (0.02) 

-0.06 (0.02) 

0.05 (0.02) 

lEP slope 



Previously Eligible slope 

School lEP 

Average home materials 

0.05 (0.01) 

-0.09 (0.01) 

-0.10 (0.02) 


Note: The numbers in parentheses represent the standard error of estimate. 


As noted above, three student variables were found to have 
significant between school differences: ESL sfafus, lEP sfafus, and 
previously eligible sfafus. The average coefficienfs (slope) for fhese 
variables were negafive (see Column 2, Table 2). Thus sfudenfs wifh 
fhese designafions had lower reading scores fhan sfudenfs wifhouf fhese 
designafions. As shown in Panel B, Table 3, fhe coefficienfs were again 
all weak (<|0.12|). Analyses were fhen conducfed fo see if fhe school 
level variables considered in fhe presenf sfudy accounfed for fhese 
school differences. In fhe case of fhe differential ESL school effect (slope), 
the negative coefficient (-0.07) for number of sfudenfs indicafes fhaf fhe 
difference (gap) in reading scores befween ESL and non-ESL sfudenfs 
was larger in larger schools than in smaller schools. This suggests that 
ESL students had relatively poorer success in larger schools than in 
smaller schools. The positive school lEP coefficient (0.11) indicates that 
the scores of ESL and non-ESL sfudenfs were closer in schools having 
more lEP sfudents fhan in schools having fewer lEP sfudenfs. None of 
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the school level variables was found to be associated with the differential 
lEP slope. Two school-level variables were found to affect the previously 
eligible slope. Schools with more lEP students had more similar scores 
between first time and previously eligible students than schools with 
fewer lEP students. In contrast, the gap between first time writers and 
previously eligible students was larger in schools in which students had, 
on average, more home literacy materials. 

Writing results. As shown in Table 2, six student status variables, two 
of the 12 home literacy survey items, and three of the five scaled survey 
items/factors were included in the model for writing at the student level. 
With two exceptions, the values of the coefficients for writing were less 
than the corresponding values of the coefficients for reading. The one 
large exception was previously eligible status, which had a much 
stronger negative association with writing scores (-0.97) than that found 
for reading scores (-0.51). Elowever, like reading, the associations 
between the student variables and writing performance were in the 
expected direction with the same one exception (creative literacy 
activities). Similarly, three of the student-status variables had the largest 
associations with reading scores. The relatively large negative 
associations of lEP status and, especially, previously eligible status with 
writing performance indicated that these students' writing scores were, 
respectively, on average 0.28 and 0.97 of a standard deviation lower than 
the average writing score. In contrast, IPRC-gifted students had writing 
scores that were approximately 0.37 of a standard deviation higher than 
the average writing score. The associations of the remaining four 
student-status variables and the scaled items/factors associated with 
literacy activities were smaller (<| 0 . 10 |). Lastly, as pointed out, 

creative literacy activities had a small negative association with writing 
performance. 

The school level results for the writing model are presented in the 
third column of Table 3. Panel A contains the school-level relationships 
with the mean writing achievement in schools (intercept). Two Level 2 
variables - average home computer use for school work (0.10) and 
average home literacy materials (0.09) - were significantly associated 
with writing achievement in the final model (Panel B, Table 3). As the 
average student use of a home computer for school work and the 




Contextual and School Factors 


789 


average student use of home literacy materials increased, the average 
writing performance in schools increased. 

Panel B confains fhe resulfs for fhe fhree sfudenf sfafus variables - 
ESL {tau = 0.09), lEP {tau = 0.06), and previously eligible ( tau = 0.31) - 
fhaf had significanf befween school variance, suggesfing fhaf fhe 
differences in wrifing scores obfained by fhese designafed sfudenfs could 
be affribufed, af leasf in parf, fo fhe schools fhaf fhey affended. The same 
fhree sfudenf variables found for reading had significanf befween school 
differences for wrifing. In fhe case of fhe ESL slope, fhe negafive 
coefficienf (-0.06) for number of sfudenfs indicafed fhaf fhe difference 
(gap) in wrifing scores befween ESL and non-ESL sfudenfs was larger in 
bigger schools fhan in smaller schools. Again, as for reading, fhis finding 
suggesfs thaf ESL sfudenfs had relafively poorer success in larger schools 
fhan in smaller schools. Again, none of fhe school-level variables was 
found fo be associafed wifh fhe lEP slope. Easily, fhe gap befween firsf 
fime wrifers and previously eligible sfudenfs was larger in schools in 
which sfudenfs had, on average, more home liferacy maferials fhan in 
schools in which fhe sfudenfs had, on average, fewer home liferacy 
maferials. 

Explained variability. The proportion of fhe sfudenf and school 
variabilify explained by fhe sef of variables included af fhe fwo levels of 
fhe model for reading and fhe model for wrifing were calculafed. Firsf, 
as nofed previously, mosf of fhe initial variation in achievemenf was 
among sfudenfs wifhin classes: 86.5 per cenf for reading and 90.0 per 
cenf for wrifing. The inifial variation af fhe school level was 13.5 per cenf 
for reading and 10.0 per cenf for wrifing. Nof unexpecfedly, fhe amounfs 
of variance fo be accounfed for af fhe sfudenf level are considerably 
greafer fhan fhe amounfs fo be explained af fhe school level (see also 
Rogers, Wenfzel, & Ndalichako, 1997; Yair, 1997). The 13 sfudenf-level 
and 8 school-level variables refained in fhe final model for reading 
accounfed for 24.3 per cenf of fhe inifial variabilify among sfudenfs and 
87.0 per cenf of fhe inifial variabilify among schools. The corresponding 
percenfages for wrifing were 27.7 per cenf and 76.3 per cenf. Alfhough 
fhe variances af fhe fwo levels of fhe final models for reading and 
wrifing were considerably less fhan fhe inifial values found for bofh 
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subjects, there was still unexplained variability, particularly at the 
student level. 

DISCUSSION 

The Ontario Secondary Schools Literacy Test (OSSLT) provides a unique 
opportunity to investigate the associations between student- and school- 
level factors with achievement as measured on a high-stakes 
examination. To the extent that high-stakes examinations, in this case a 
graduation requirement, result in increased student effort and more 
accurate achievement scores, the results of fhe OSSLT can be viewed as 
better indicafors of achievemenf fhan dafa obfained from low-sfakes 
examinafions. The resulfs of fhe analyses provide an indicafion of 
relevanf sfudenf- and school-level variables fhaf are associafed wifh 
reading and wrifing achievemenf as measured by a high-sfakes 
achievemenf fesf. Through fhe idenfificafion of such associations, if 
becomes possible fo examine policy implications. 

As found in previous research, fhe majorify of fhe variability for bofh 
fhe reading and wrifing achievemenf scores was befween sfudents rafher 
fhan befween schools (e.g., Ma & Klinger, 2001; Raudenbush & Bryk, 
2002; Rogers, ef al., 2000; Rogers, Wenfzel, & Ndalichako, 1997; Yair, 
1997). Several sfudenf-level variables were associafed wifh higher 
achievemenf, buf fhe amounf of unexplained sfudenf-level variance 
remained subsfanfial. Less fhan 30 per cenf of fhe sfudenf-level 
variabilify in achievemenf was accounfed for by a sef of available sfudenf 
variables. Sfudenf sfafus (ESL, lEP, gifted, and previously eligible) had 
fhe largesf associations wifh fhe achievemenf variables. Literacy related 
variables had small associations with both reading and writing. Such 
findings are discouraging from a policy perspecfive because fhey 
indicafe fhaf sfudenf contexfual variables continue to have the most 
identified influence on achievemenf. Elowever, because fhe majorify of 
sfudenf-level variability was not accounted for in fhe final models, fhe 
resulfs also indicate that these contextual variables are not sufficient to 
explain the variability in student performance. As noted previously, with 
one exception, the associations tended to be smaller between the student 
variables and writing performance fhan befween fhe student variables 
and reading performance. The one nofable exception was fhe previously 
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eligible variable that had a much stronger negative association with 
writing performance than with reading performance (-0.97 vs. -0.51). 
This suggesfs fhaf previously eligible sfudenfs had relatively more 
difficulfy wifh writing fhan wifh reading as measured by fhe 2003 
OSSLT. If would be worfhwhile fo examine if fhis frend confinues across 
fufure adminisfrafions. 

Alfhough fhe relafively low reliabilify of fhe creafive liferacy 
activities facfor indicafes caufion, fhe small buf consisfenf negative 
associations befween creafive liferacy acfivifies and reading and writing 
are also worfhy of furfher sfudy. This finding may suggesf fhaf sfudenfs 
who focus more of fheir attention on engaging in more creafive liferacy 
acfivifies may have more difficulfy on fhe liferacy examination. If is 
possible fhaf fhese sfudenfs represenf a unique sub-group who do nof 
perform well on a fradifional measure of liferacy. 

Alfhough mosf of fhe variabilify in achievemenf was found fo be 
befween sfudenfs rafher fhan befween schools, school-level associations 
were found. For example, schools wifh sfudenfs who made more use of 
fhe compufer af home fo assisf wifh fheir school work or had increased 
amounts of liferacy relafed home maferials also had sfudenfs wifh higher 
wrifing scores. Of inferesf was fhe lack of impacf of fhe school-level SES 
measure on achievemenf. In fhe presence of ofher school-level liferacy 
facfors and demographic variables, SES did nof appear fo have a 
significanf associafion wifh eifher reading or wrifing performance. If 
may be fhaf fhe presence of variables such as average home computer 
use or the average home materials may have diminished the association 
between SES and achievement (see Rogers et al., 2000). Elowever, the SES 
variable did not have a large association with either reading or writing 
performance when considered alone. This latter finding is somewhaf 
surprising, and may be affribufed fo fhe way SES was measured. 
Unforfunafely, direcf and indirecf sfudenf-level SES informafion was nof 
available, fhereby making if more difficulf fo defermine fhe impacf of 
fhis facfor on achievemenf on fhis high-sfakes liferacy fesf. Lasfly, fhe 
finding fhaf fhe relationship of ESL sfafus, lEP sfafus, and previously 
eligible sfafus wifh reading and wifh wrifing depended on fhe school fhe 
sfudenfs affended deserves furfher examination. Whaf facfors ofher fhan 
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school size and percentages of special status students can account for the 
differential levels of association? 

The results of the present study are somewhat disappointing in that 
the amount of variance explained was not large, particularly at the 
student level where most of the variability resides. Part of the problem in 
the present study may be due to the dichotomous nature of the student 
survey data, which served to constrain the relationships modeled. 
Further, the relatively small number of items used in the survey coupled 
with the "yes" or "no" response format made it difficult to identify stable 
relevant factors. The moderately low reliabilities of the two student-level 
factors, creative literacy activities and home literacy materials, illustrate 
this problem. A second problem may be the inability to include class as a 
level in the multi-level model. Rogers et al. (2000), for example, found 
that the variability among classes was greater than the variability among 
schools. The OSSLT is designed to represent literacy achievement across 
subject areas and is not viewed as being cormected to specific courses or 
teachers. Hence it may be argued that teacher-level effects are not 
relevant. However, because data are collected only at the student and the 
school levels, it is not possible to determine any significant class- 
/teacher-level effects associated with student achievement. A third 
problem, which is related to the second, is the absence of other variables 
that may influence performance: internal and external funding, school 
organizational structure and operation, school leadership, staff 
qualifications, nature of instructional support, quality of professional 
development and professional development opportunities, parent 
involvement and support, and, importantly, classroom processes. 
Inclusion of these variables may very well account for a substantial 
proportion of the variance in achievement measures. 

Given the similarity in the results of the current study with other 
previous studies, the information obtained through the use of high- 
stakes examinations may not be substantially different than that 
obtained using low-stakes test programs (e.g., Ma & Klinger, 2001; 
Rogers et al, 2000). The amount of score variability was not all that 
different from that found in other research in which low-stakes 
examinations were used. Further research is needed to determine if 
similar results occur across years of the examination program and with 
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similar assessment programs having lower associated test consequences 
for students. Such studies should provide evidence regarding the 
associated stakes of an assessmenf program required fo obfain accurafe 
associafions befween confexf, pracfice, and achievemenf. 

CONCLUSION 

The qualify of public education is a key confribufor fo fhe well being of 
Canadian society. The relationships between school system traits and 
schooling outcomes are of basic interesf and significance fo fhe 
educational policy communify and educafional sfakeholders. The 
foundafion for fhis inferesf lies in examining fhose variables and facfors 
fhaf can be influenced by policy makers and influence fhe key oufcomes 
of schooling (Kermedy, 1999a). The mosf commonly used indicator of 
educafional performance will continue to be student achievement on 
centrally administered assessments of learning oufcomes. If is equally 
importanf fo collecf ofher relevanf information fhaf is associated wifh 
fhis achievemenf if learning is fo be enhanced and fhe currenf gaps 
befween sub-groups of sfudenfs closed. Unforfunafely, fhe hope for 
explanatory models of complex systems such as education remains 
elusive (Lindblom, 1990). Af besf, such models are likely fo be 
enlighfening, allowing incremenfally increasing undersfandings of such 
dynamic sysfems (Kermedy, 1999b). For fhese models fo expand 
undersfandings of educafional sysfems, fhe analyses and models musf be 
school-relevanf and conducfed over fhe long term. If seems unlikely fhaf 
one model will suffice. Thus fhe research mosf appropriafe fo fhis field 
has to be of a complex, dynamic, and susfainable nafure, and findings 
have fo be fully communicafed and critically analyzed by fhe policy 
communify to feed furfher iferafions of research. The findings of fhe 
presenf sfudy, alfhough limifed, illusfrafe fhe analyses needed and poinf 
fo fhe need fo expand fhe nafure of influential variables in fhe data 
collection design. 
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NOTES 


^ Students who have been unsuccessful or have been deferred from 
writing the OSSLT may complete the Ontario Secondary School Literacy Course 
(OSSLC) to meet this graduation requirement. 

^ In 2005 - 2006, the two components were combined to form a single 
literacy score. The test is now administered in late March. 

3 The format and procedures associated with the OSSLT have changed 
slightly over time. 
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